Partitioning Input Space for Control-Learning

This paper considers the eeect of input-space partitioning on reinforcement learning for control. In many such learning systems, the input space is partitioned by the system designer. However, input-space partitioning could be learned. Our objective is to compare learned and programmed input-space partitionings in terms of the overall system learning speed and proociency achieved. We present a system for unsupervised control-learning in temporal domains with results for both programmed and learned input-space partitionings. The trailer-backing task is used as an example problem. Many classic control-learning systems, such as Michie and Chambers' BOXES 11] and Barto, Sutton, and Anderson's ASE/ACE system 2], rely on the partitioning of a space of continuous input variables into a xed number of discrete regions. More recent systems, such as Fuzzy BOXES 18], have blurred the boundaries between the input regions but nonetheless rely on a partitioning of the input space. To use these systems, researchers have typically partitioned the space manually, prior to the application of the learning system. In these cases, the designer must analyze the problem to discover a suitable partitioning, or face a trade-oo between a ne partitioning that permits accurate approximation of complex functions or a gross partitioning that allows for rapid learning. The Self-Organizing Neural Network with Eligibility Traces (SONNET) scheme introduced by Hougen 4] is a general paradigm for the construction of connectionist networks that learn to control systems with a temporal component. In order to form mappings from input parameters to output responses, the system discretizes the input space by learning a partitioning of it, and learns an output response for each resulting discrete input region. SONNET systems have separate subsystems for learning input and output. The input subsystem learns input-space partitionings through self-organization and the output subsystem learns responses through the use of eligibility traces. Both input and output learning make use of topological ordering of the neural elements and associated neighborhoods, as in Kohonen's Self-Organizing Topological Feature Maps 8]. 1.1 Topology and neighborhoods Each SONNET subsystem consists of one or more artiicial neural networks. For each network there is a topological ordering of the neural elements that remains constant as the network learns. Each neural element is assigned an integer tuple of the same dimensionality as the network that uniquely deenes its coordinates in topology space. The existence of a network topology allows for the deenition of a distance function for the neural elements. This is …